A Parallel and Distributed Approach for Finding Transitive Closures of Data Records: A Proposal

In this paper, we propose an approach to find transitive closures on large data sets in distributed (i.e., parallel) environment. Finding transitive closures of data records is a preprocessing step of a two-step approach to data quality control, such as data accuracy, redundancy, consistency, currency and completeness. The objective of finding transitive closures is to reduce the number of records to be considered in the second step, from a whole data source having hundreds of millions to billions of records to the range of hundreds to thousands. To process hundreds of millions to billions of records, an efficient approach is essential that works in distributed environment. As a part of this approach, this paper presents an efficient distributed algorithm for solving distributed transitive closure problem on large data sets. Due to huge volumes of data, many real world applications are in need of fast and efficient approaches for data analysis and data mining. However, data cleansing, which precedes data analysis and data mining, is becoming the center of research interest in recent years. As a part of it, the process of finding transitive closures (i.e., finding all related records) is the main goal of this paper. The computation of transitive closures of data records has two related but independent activities. One is to determine if two records are related based on some definition of relatedness for virtually all pairs of records. The other is to identify the transitive closures based on the relations. A particular definition of a relation between two records is considered in this paper. And the proposed parallel and distributed approach takes advantage of this definition to improve the performance of transitive closure computation in grid computing environment.